Very interesting! I did encounter slow prompt processing performance for GLM 4.5 Air on pure CUDA at -ub 512 previously, so I'm curious how much this helps.

I'm having some trouble getting this PR to compile on Windows. It seems to be encountering the same error building multiple mmq instance files:

C:\ML\ik_llama.cpp\ggml\src\ggml-cuda\template-instances\../mmq_id_common.cuh(240): error : identifier "__nv_cvt_e8m0_t
o_bf16raw" is undefined [C:\ML\ik_llama.cpp\build\ggml\src\ggml.vcxproj]
        const nv_bfloat16 e = __nv_cvt_e8m0_to_bf16raw(x);
                              ^

@DocShotgun Thanks! There was the fp8 header include missing. Can you try now?

@DocShotgun Thanks! There was the fp8 header include missing. Can you try now?

Built successfully now!

Here are some preliminary numbers on Windows 11 with RTX PRO 6000 96gb on GLM-4.5-Air-Q5_K_S. Ran the same tests that you did above. It's a HUGE speedup on Windows lol.

ik_llama.cpp PR vs ik_llama.cpp main 50f7119dfd15325c6889927499e4233f7453cb44:

./llama-bench -m "C:\ML\GGUF\GLM-4.5-Air-Q5_K_S.gguf" -ngl 999 -fa 1 -fmoe 1 --mmap 0 -t 1 -n 0 -p 4096 -b 4096 -ub 128,256,512,1024,2048,4096

I compared it to current mainline llama.cpp as well (with Johannes's original PR merged).

ik_llama.cpp PR vs llama.cpp main 34bdbbd7c2b70b848718e95bc48010f6aecd2816:

./llama-bench -m "C:\ML\GGUF\GLM-4.5-Air-Q5_K_S.gguf" -ngl 999 -fa 1 -fmoe 1 --mmap 0 -t 1 -n 0 -p 4096 -b 4096 -ub 128,256,512,1024,2048,4096
(same thing for main llama.cpp but with no -fmoe)

No real changes for hybrid inference. Something made PP fall when I went from batch 1024 to 512, previously it was over 100 on qwen-235b and now it's 66, but I don't think that it was this commit.

No real changes for hybrid inference. Something made PP fall when I went from batch 1024 to 512, previously it was over 100 on qwen-235b and now it's 66, but I don't think that it was this commit.

I think something happened in the CUDA graphs PR, but I have not taken the time to measure commits yet and find it.

I may also be mistaking qwen 512 perf for GLM perf. I tried rolling back to 23fe18c with no change. I'll try to roll back to before cuda graphs later today.

Because @DocShotgun brought up a comparison with mainline llama.cpp: for the MoE models I can run fully offloaded to my 16 GB RTX-4080 GPU, ik_llama.cpp is still faster (but yes, Johannes has been able to close the gap quite a bit since April, when the llama.cpp performance improvement initiative kicked in). Here is what I get for 3 different models:

No real changes for hybrid inference.

Of course not. For hybrid inference such small batches may not even be offloaded to the GPU (batch size must be greater than 32 * num_total_expert / num_active_experts for that to happen). But even if they do get offloaded, for such small batch sizes total processing time is totally dominated by the time it takes to copy the tensors to VRAM. For large batches, where data transfer time is a smaller fraction of the overall time, the improvement is much less (and it eventually completely goes away), so performance improvements will be none to very small.

Something made PP fall when I went from batch 1024 to 512,
may also be mistaking qwen 512 perf for GLM perf

Yes, it can be very useful to take notes, so one does not live with the concept that something got broken along the way.

Because @DocShotgun brought up a comparison with mainline llama.cpp: for the MoE models I can run fully offloaded to my 16 GB RTX-4080 GPU, ik_llama.cpp is still faster (but yes, Johannes has been able to close the gap quite a bit since April, when the llama.cpp performance improvement initiative kicked in). Here is what I get for 3 different models:

Do you think mainline llama.cpp still has some other Windows-specific optimization that hasn't been ported over yet?

Funny enough, I keep notes with benchmarks but the file is getting unwieldy. Got a small 512 bump from 32 * num_total_expert / num_active_experts I'll have to clear the cache to get a proper reading on 1024 as loading different models in/out of memory has a detrimental effect after a while.

This PR might have provoked a perplexity bump on GLM 4.5 Air. I inquried about a perplexity bump on the recent main, as I was trying your commit allowing to quantize ffn_gate_imp, the PPL was higher than expected, including on the source model (the following quant in my 'expose').

I rolled back from main to isolate the problem on PR 728. OS used, Win 11. Full Cuda offload on 3 Ampere GPUs.

Q:\LLAMA_IK_GLM2>llama-perplexity -m X:\text-generation-webui\models\zai-org_GLM-4.5-Air-bf16-iMat-IQ4_XS_L-00001-of-00005.gguf -f wiki.test.raw --parallel 1 -ngl 150 -b 512 -mg 0 -ts 18,18,12 --no-mmap -fa --override-kv glm4moe.expert_used_count=int:7

llama_model_loader: - type  f32:  331 tensors
llama_model_loader: - type q4_K:    3 tensors
llama_model_loader: - type q5_K:   93 tensors
llama_model_loader: - type q6_K:   53 tensors
llama_model_loader: - type iq4_nl:   45 tensors
llama_model_loader: - type iq4_xs:  136 tensors
llama_model_loader: - type q6_0:  142 tensors

Before this PR 728 :

Final estimate: PPL = 4.6508 +/- 0.02854 / orig before Muh PR.

llama_print_timings:        load time =   29100.68 ms
llama_print_timings:      sample time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings: prompt eval time =  650424.94 ms / 289280 tokens (    2.25 ms per token,   444.76 tokens per second)
llama_print_timings:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings:       total time =  659193.06 ms / 289281 tokens

After this PR 728 :

Final estimate: PPL = 4.6772 +/- 0.02874 / after Muh PR.

llama_print_timings:        load time =   29011.64 ms
llama_print_timings:      sample time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings: prompt eval time =  456817.57 ms / 289280 tokens (    1.58 ms per token,   633.25 tokens per second)
llama_print_timings:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings:       total time =  465547.22 ms / 289281 tokens

I made a compromise for myself between speed and quality by shelving 12 PRs, making a clean rollback.

Here's the modified branch :

https://github.com/Nexesenex/ik_llama.cpp.nxs/tree/Rolled-back-to-pre-PR728

final estimate: PPL = 4.6544 +/- 0.02857 / branch rolled back from main until pre-PR 728.

llama_print_timings:        load time =   29011.65 ms
llama_print_timings:      sample time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings: prompt eval time =  513583.20 ms / 289280 tokens (    1.78 ms per token,   563.26 tokens per second)
llama_print_timings:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings:       total time =  522307.88 ms / 289281 tokens

Disabling MMAD on this branche helps quality furthermore:

Final estimate: PPL = 4.6511 +/- 0.02854 / branch rolled back from main until pre-PR 728. -no-mmad

llama_print_timings:        load time =   29681.95 ms
llama_print_timings:      sample time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings: prompt eval time =  530091.86 ms / 289280 tokens (    1.83 ms per token,   545.72 tokens per second)
llama_print_timings:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings:       total time =  538881.31 ms / 289281 tokens

@Nexesenex

What PPL do we get with mainline llama.cpp? Or is this a mix that cannot be computed with llama.cpp?

Btw, you don't need to roll back 12 PR and all that. To arrive at the behavior prior to this PR, all you need to do is change

to

    if (false && src1->ne[2] <= 32*src0->ne[2]

OK, I can confirm the issue exists in mainline llama.cpp as well. Here is a table with perplexities for a few models. PPL has been computed with today's llama.cpp build (build: 6967 (b7f9010d2)). PPL* is also with today's build, but disabling the effect of mainline PR 15525, on which this PR is based, by changing

        if (ggml_cuda_should_use_mmq(src0->type, cc, ne12)) {
            ggml_cuda_mul_mat_q(ctx, src0, src1, ids, dst);
            return;
        }

on lines 2285-2288 in ggml-cuda.cu to

        if (false && ggml_cuda_should_use_mmq(src0->type, cc, ne12)) {
            ggml_cuda_mul_mat_q(ctx, src0, src1, ids, dst);
            return;
        }

The PPL increase appears to be systematic. Differences of 0.4% are well beyond the typical range caused by numerical round off. Hence, it is likely that there is an issue in mainline PR 15525, so pinging @JohannesGaessler as the author of 15525.

Btw, you don't need to roll back 12 PR and all that. To arrive at the behavior prior to this PR, all you need to do is change

ik_llama.cpp/ggml/src/ggml-cuda.cu

Line 2688 in 575e2c2

if (src1->ne[2] <= 32*src0->ne[2] &&

to

    if (false && src1->ne[2] <= 32*src0->ne[2]

I suspected that PR first, and saw that line, but then I tried to undo the commit and saw conflicts, so I concluded that it could be a more complex issue and that changing a trigger might not cut the deal, so I rolled-back everything to be absolutely certain of myself. I didn't check on mainline, though, I was sleepy. :D

My variances are in the same order of magnitude than yours, and they indeed went way beyond the 0.01% or so which is "acceptable".

Anyway, showtime for the big guys! ^^

@Nexesenex See #910. With this, using -cuda mmq-id-size=0 will undo the effect of this PR.

Thank you for notifying me. I am seeing an issue specifically on my RTX 5090, with no other GPU being affected. To be clear, the hardware you were using is pre-Blackwell, right? (Could be the same bug, but manifesting differently depending on SM count.)

@JohannesGaessler

I'm using RTX-4080. @Nexesenex is using 3090s IIRC.

I am seeing an issue specifically on my RTX 5090, with no other GPU being affected.

You mean you computed perplexity on a bunch of MoE models, and the result is the same pre- and post MUL-MAT-ID optimization on all of the many GPUs that you have, except on the 5090?

No, I mean that on my RTX 5090 (which I didnt yet have back then) there is a very clear problem with ppl increasing by 50% so I'll debug that first. I think it has to do with the SM count not being a power of 2 which would affect all GPUs in question.

I'm not a GPU guru, but the SM count being a power of 2 is not something I would expect. It is 76 on my 4080, and 82 on the 3090 (which is a pretty popular choice among LLM enthusiasts).

Anyway, I hope you will find the issue. Thanks for looking into it!

The reason I suspect it is simply because for my implementations of stream-k decompositions that has historically been a frequent source of bugs. The testing I did was simply GraniteMoe with master (+50% ppl vs. RTX 4090 with 128 SMs), with your patch (fixes ppl), and with a patch that overrides the SM count (from the perspective of ggml) to 32 (also fixes ppl). The issue has persisted since back when I implemented this feature on the upstream repository.